Gesture-Based Computer Interface Using Hand Gestures and Eye Blink Detection

Abstract

Conventional human-computer interaction relies on physical input devices such as a mouse and keyboard, which is not always convenient, hygienic, or accessible to every user. This paper presents a touchless, webcam-based computer interface that allows a user to control common mouse and navigation functions using hand gestures and eye blinks, without any dedicated sensor or wearable hardware. The system captures live video using OpenCV and processes each frame with Media Pipe Hands and Media Pipe Face Mesh to obtain twenty-one hand landmarks and dense facial landmarks in real time. The hand landmarks are used to move the on-screen cursor, and to recognise pinch, fist, two-finger, and multi-finger gestures that are mapped to clicking, dragging, scrolling, zooming, volume adjustment and swipe-based page navigation. The facial landmarks around each eye are used to compute the Eye Aspect Ratio (EAR), which is thresholder and timed to distinguish short and long blinks of the left and right eye, each of which is mapped to a distinct mouse action. Detected gestures and blink events are translated into system-level mouse and keyboard commands using PyAutoGUI. An on-screen heads-up display further allows the user to enable or disable individual interaction modes at run time through a pinch-based menu. The proposed system demonstrates that a standard webcam, combined with pretrained landmark-detection models, is sufficient to build a functional, low-cost, touchless computer interface with applicability to accessibility-oriented and hygienic human-computer interaction scenarios.

Introduction

The text presents a touchless computer-control system that combines hand gestures and eye blinks to control a computer using only a standard webcam. The system is designed to reduce dependence on physical mouse and keyboard interaction while improving hygiene, accessibility, and hands-free computing.

Main Objective

The primary goal is to develop a real-time, webcam-based human-computer interface without requiring specialized hardware, depth sensors, or custom-trained machine-learning models. The system combines:

• OpenCV for webcam capture and image processing.
• MediaPipe Hands for detecting 21 hand landmarks.
• MediaPipe Face Mesh for detecting facial and eye landmarks.
• Eye Aspect Ratio (EAR) for identifying eye blinks.
• PyAutoGUI for converting recognized gestures into actual mouse and keyboard actions.

System Working

Each webcam frame is processed simultaneously for hand and eye information. The hand landmarks are used to identify finger positions, pinches, finger counts, fists, and hand movements. The eye landmarks are used to calculate EAR and determine whether the user is blinking.

The index fingertip controls the cursor, while different finger combinations and movements trigger different computer operations. Cursor movement is smoothed using an exponential moving-average filter to make interaction more stable.

Gesture and Blink Controls

The system supports a wide range of computer commands, including:

• Index-finger movement → Cursor movement
• Thumb-index pinch → Left click
• Thumb-middle pinch → Drag
• Fist → Lock cursor
• Two fingers → Scrolling
• Fast vertical movement → Air scrolling
• Pinch-distance change → Zoom
• Fast horizontal movement → Swipe navigation
• One, two, or three fingers → Different mouse clicks
• Specific pinch gestures → Volume control
• Short left-eye blink → Left click
• Long left-eye blink → Double click
• Short right-eye blink → Right click
• Long right-eye blink → Click and hold

Related Work

Earlier virtual-mouse systems mainly relied on colour segmentation, skin detection, fingertip tracking, convex-hull analysis, and object tracking. More recent approaches have adopted MediaPipe-based landmark detection because it provides efficient real-time hand and face tracking without requiring specialized hardware.

The proposed system builds upon these developments by integrating hand gestures and eye-blink interaction into one unified interface.

Conclusion

This paper presented a touchless, webcam-based computer interface that combines hand-gesture recognition and eye-blink detection to control common mouse and navigation functions. Using OpenCV for video capture, MediaPipe Hands and MediaPipe Face Mesh for real-time landmark detection, and PyAutoGUI for system-level control, the system supports cursor movement, clicking, dragging, scrolling, zooming, volume control, and swipe navigation through hand gestures, together with click, double-click, right-click and click-and-hold actions through short and long blinks of either eye. The system further provides an on-screen HUD that lets a user toggle individual gesture features at run time. The proposed approach shows that a practical, low-cost, touchless computer interface can be built using only a standard webcam and pretrained, open-source computer-vision models, without the need for specialised hardware or a custom-trained gesture-recognition network. Future improvements, including adaptive thresholds, multi-hand support, and a formal user study, can further improve the robustness and usability of the system.

References

[1] G. Bradski, “The OpenCV Library,” Dr. Dobb\'s Journal of Software Tools, 2000. [2] C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. Yong, J. Lee, W.-T. Chang, W. Hua, M. Georg, and M. Grundmann, “MediaPipe: A Framework for Building Perception Pipelines,” arXiv:1906.08172, 2019. [3] F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, “Media Pipe Hands: On-device Real-time Hand Tracking,” arXiv:2006.10214, 2020. [4] Y. Kartynnik, A. Ablavatski, I. Grishchenko, and M. Grundmann, “Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs,” in Proc. CVPR Workshop on Computer Vision for Augmented and Virtual Reality, 2019. [5] S. S. Rautaray and A. Agrawal, “Vision based hand gesture recognition for human computer interaction: a survey,” Artificial Intelligence Review, vol. 43, no. 1, pp. 1–54, 2015. [6] T. Soukupová and J. ?ech, “Real-Time Eye Blink Detection Using Facial Landmarks,” in Proc. 21st Computer Vision Winter Workshop (CVWW), 2016. [7] A. Sweigart, “PyAutoGUI Documentation,” Read the Docs. [Online]. Available: https://pyautogui.readthedocs.io/ [8] K. H. Shibly, S. K. Dey, M. A. Islam, and S. I. Showrav, “Design and Development of Hand Gesture Based Virtual Mouse,” in Proc. 1st Int. Conf. Advances in Science, Engineering and Robotics Technology (ICASERT), Dhaka, Bangladesh, 2019, pp. 1–5, doi: 10.1109/ICASERT.2019.8934612. [9] D. M. S. Reddy, S. Kukkamudi, R. Kunda, and T. Mamatha, “Virtual Mouse Using Hand Gesture,” in Proc. Int. Conf. Knowledge Engineering and Communication Systems (ICKECS), 2022, pp. 1–5, doi: 10.1109/ICKECS56523.2022.10060367. [10] K. S. Varun, I. Puneeth, and T. Prem Jacob, “Virtual Mouse Implementation Using Open CV,” in Proc. 3rd Int. Conf. Trends in Electronics and Informatics (ICOEI), 2019, doi: 10.1109/ICOEI.2019.8862764. [11] T. Tsai and Y.-R. Tsai, “Embedded Virtual Mouse System by Using Hand Gesture Recognition,” in Proc. IEEE Int. Conf. Consumer Electronics – Taiwan (ICCE-TW), 2015, doi: 10.1109/ICCE-TW.2015.7216939. [12] M. Ranawat, M. Rajadhyaksha, N. Lakhani, and R. Shankarmani, “Hand Gesture Recognition Based Virtual Mouse Events,” in Proc. 2nd Int. Conf. for Emerging Technology (INCET), Belagavi, India, 2021, pp. 1–4, doi: 10.1109/INCET51464.2021.9456388. [13] V. V. T. Reddy, T. Dhyanchand, and G. V. Krishna, “Virtual Mouse Control Using Colored Fingertips and Hand Gesture Recognition,” in Proc. IEEE India Council International Conference (HYDCON), Hyderabad, India, 2020, pp. 1–5, doi: 10.1109/HYDCON48903.2020.9242677. [14] M. Shetty, C. Daniel, M. Bhatkar, and O. Lopes, “Virtual Mouse Using Object Tracking,” in Proc. 5th Int. Conf. Communication and Electronics Systems (ICCES), 2020. [15] S. R. Chowdhury, S. Pathak, and M. D. A. Praveena, “Gesture Recognition Based Virtual Mouse and Keyboard,” in Proc. 4th Int. Conf. Trends in Electronics and Informatics (ICOEI), 2020, pp. 585–589.

Copyright

Copyright © 2026 Shyalaja L N, Prajwal P C , Siddhanth Sasture , Keerthana G V, Srishti K R. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.